When video is streamed over the Internet and played back through a Web browser or media player, the video is delivered in digital form. Digital video is also used when video is delivered through many broadcast services, satellite services and cable television services. Real-time videoconferencing often uses digital video, and digital video is used during video capture with most smartphones, Web cameras and other video capture devices. Digital video is also used for technologies such as virtual reality (“VR”) and augmented reality (“AR”), whether video is played back in a head-mounted display, mobile device, or other type of device.
Panoramic video is video in which views in multiple directions around a central position are recorded at the same time. The recorded video can include image content in every direction, or at least image content in every direction in a 360-degree circle around the central position, as well as at least some image content above the central position and at least some image content underneath the central position. Panoramic video is sometimes called 360-degree video, immersive video, or spherical video. Panoramic video can be captured using an omnidirectional camera or a collection of multiple cameras pointing in different directions. For modern-day applications, panoramic video is processed in digital form during stages of creation, editing, and delivery, as well as stages of reconstruction and rendering for playback.
During playback, a viewer typically can control a view direction relative to the central position, potentially changing which section of the panoramic video is viewed over time. In some systems, a viewer can also zoom in or zoom out. When panoramic video is rendered for display, the section of the panoramic video that is viewed may be projected to a flat image for output. For a mobile device or computer monitor, a single output picture may be rendered. For a head-mounted display (or mobile device held in a head-mounted band), the section of the panoramic video that is viewed may be projected to two output pictures, for the left and right eyes, respectively.
When a playback tool reconstructs and renders panoramic video, resources may be wasted retrieving and reconstructing image content that is not viewed. For example, memory may be used to store sample values for areas of the panoramic video that are not viewed, and processing cycles may be used to determine the non-viewed sample values and their locations at different stages of processing.
To use fewer resources, a playback tool may retrieve and reconstruct only part (not all) of the panoramic video. For example, considering the view direction and zoom factor for a viewer, the playback tool may retrieve encoded data and reconstruct panoramic video just for those sections of the panoramic video that are visible. In this way, the playback tool may save memory, processing cycles, and other resources while correctly rendering the visible sections of the panoramic video. If the view direction or zoom factor changes, however, the playback tool may not have image content needed to correctly render sections of the panoramic video that should be visible. Playback may freeze or stall until the playback tool can recover by retrieving encoded data and reconstructing panoramic video for the newly visible sections.